Sixel + Kitty support for "fine resolutions" - #42

Very interesting! What is your go-to terminal emulator for testing sixel support?

I've only toyed with a few so far, see last column "graphics" of this chart https://ucs-detect.readthedocs.io/results.html, plan to add "Graphics" column in README compatibility table as I go through them.

I think I got resizing working well, to determine size and to clear old image, not sure about mode switching (F5-F8), needs testing.

@vxgmichel I have finished a grueling amount of refinement, it was much harder than I imagined, but sixel and kitty support are working well and tested on all linux terminals, I'll check windows and macos soon, video: https://www.jeffquast.com/sixel-and-kitty-graphics.mp4

• There is an "autoscaler" that is on by default, it reduces the scale if FPS drops below 55 in the first 99 seconds.
  • some terminals, iTerm2 and Contour (HEAD) in particular, are very "boggy" and "freeze up" when you resize the windows, and the autoscaler reacts too rapidly, quickly sizing down the scale, I need to add a 2 second "cool down" period, iTerm2 really "locks up" for a whole 2 seconds when resizing !
• some terminals require what I call "blitless" modes -- that they are not able to take advantage of the highly compressible "transparency pixel" in sixel, drawing black instead. It could be said this transparency is part of the ancient standard, but proper implementation is in dispute at times, there are 4 terminals that require an exception
• and similarly for kitty graphics, except through a different methodology of image ID's of KEY_FRAME and DELTA's by "dirty rectangles", again, strict interpretation of standard should allow this to work correctly but ghostty and rio do not implement them correctly and have exceptions
  • This was very hard to get right! But it pays off very well in bandwidth resources for all other supporting terminals
  • Note that although kitty graphics protocol supports scaling images as a built-in, "scale image to this text area", it is bilinear and blurry, so we take this very long route to ensure integer scale.
    • I tried to predict, "If I 'padd' the image with pillar and letterboxes to align in perfect integer scale of cell text size, the built-in scaler should be able to scale without blur", but I couldn't get the maths to work it out especially for the "dirty rectangle" blitting, it requires more study, I may still try before removing this PR from draft mode

also,

There are new environment variable-enabled options GAMBATERM_PROFILE_DIR and GAMBATERM_DUMP_FRAMES, this has helped me write the tests and do troubleshooting of the blitter, graphics modes, etc, so I will keep them in, though they are a bit cumbersome to use, I depended on them often.

There are also some environment variables for limits on maximum scale, which approximates about the same size as "text mode" scale for most terminals -- terminals in "BAD_TEXT", group with font corruption never switch to text mode (unless backspace is pressed) and so we must set a reasonable size limit there. mlterm is also poor performing and good emulator to test "auto scaling" with.

https://www.jeffquast.com/sixel-and-kitty-graphics.mp4

Wow that looks great! What a crazy amount of work you pulled off oO

I'm quite busy at the moment, so I apologize for not giving this PR the attention it deserves.

I'm also curious about the autoscaler, do you test it with CPR sync enabled? Because as far as I remember, the FPS can drop for two reasons:

1. either the main emulator loop cannot keep up for some reason and the best thing we can do is drop frames and hope it will help (that's the shift/shifting detection logic)
2. or CPR sync is enabled and we detect that either the network or the remote terminal cannot keep up with the amount of data we're sending

Now, the typical CPU usage of the emulator is 4% and the video (i.e the blitter) is 1%, so at 1x speed, scenario 1 is pretty unlikely, unless the remote terminal or network is very slow and a couple of MB of data are already waiting to be processed in the client and/or server TCP buffers, causing the run loop to block on the TCP write. In that case however, a couple of MB is quite a lot of frames so it's likely that the game already became unplayable, which is why I'd like to enable CPR sync default, with the solution described in #31:

A good solution in this case would be to accept a couple of frames of delay. This can be easily achieved counting the number of CPR that are currently unacknowledged. Once this number is too high (e.g more than a configurable parameter, defaulting to 3), then we skip sending frames until a CPR response is received.

Do you see things the same way? This subject is getting a bit too complex to easily reason about it, I should document this better at some point.

Also we've been discussing many rendering strategies (including sextants and octants), do you think we can always come up with a way to properly decide what's the best one for a given terminal in a given context? Or should we allow users to switch between them? I'm still confused about what to do with the UI, and what the user experience should be. Ideas are welcome :)

I will test/integrate with --cpr-sync, I've been ignoring it so far, in fact looks like it freezes up the video for some reason :)

When very busy with encoding graphics at scale as high as 11x, CPU usage of gambaterm's video encoding can also be as high as 70%, bandwidth as high as 3,000KB/s. Instead of dropping any frames, I've been lowering the scaling of graphics, but dropping video frames or going half-speed might be better, I'll experiment with it.

As for Octants and Sextants, I think Octants might be avoided because we can't reliably detect whether it is supported,

• although there is a new protocol that allows us to ask, it is a small part of a very large and complex specification found only in rio terminal, and I don't expect to see it adopted outside of rio, as they provide no "base library" or reference implementation (like libghostty or libvte does), rio has numerous rendering issues, 1, 2, and 3 right now after a large refactor of their graphics layer -- although kitty graphics works it is also nerfed, requiring FORCE_KITTY_BLITLESS = ("rio",), it's a struggle to use rio graphics rendering at all.
• Sextants are reliable but the aspect ratio is wrong, I'm not really happy with either of them. Maybe it can be fixed, using the PIL method of scaling you demonstrated before, but I worry about loss off fidelity and multicolor issues. Might still be worth it, though.

Auto-switching between text and graphics protocols is good UX, that it always fits in the window and always has correct colors and aspect ratio. The only issue is that it would be really hard to communicate to the user, "if you make your window large enough, and font small enough, it will improve the FPS and lower bandwidth, or, that If you make your font smaller, it will switch to text mode and the picture will get larger ?! I'll do my best to document about it in the README

I did have backspace/delete key cycling between available graphics modes, but I removed it as I tried to clean up and simplify, because I felt there was no need to switch modes other than for developer testing (offered by GAMBATERM_FORCE_SIXEL=1 or GAMBATERM_FORCE_KITTY=1 env values)

although there is a new protocol that allows us to ask

FYI, mintty has a fairly simple protocol for detecting glyph support, using an OSC 7771 sequence. I don't think it's really practical for querying large ranges of glyphs, although you could probably get by with just querying the first and last character of the glyph set you want to use.

I also don't think any other terminals support it (other than mintty forks), but if there was significant demand for this kind of functionality, it probably has more chance of being adopted.

ETA: If you want to do something with sextants, it's worth noting that you could do that more efficiently with a DRCS character set, support for which can easily be detected with a DA1 query.

Thanks @j4james, I checked on DRCS of all terminals tested by https://ucs-detect.readthedocs.io/results.html, 5 report support, but I think 3 of those are in error. I hope I'm wrong but the support seems low from first look

• Only mlterm and Windows Terminal report support of both 7/DRCS and 4/Sixel.
• mintty, AbsoluteTelnetSSH, or SecureCRT report support for 7/DRCS and not 4/Sixel. This seems suspicious, as DRCS "soft fonts" are encoded as sixel data

@jquast Note that sixel is both a binary-to-text encoding (akin to base64), as well as the name of an image protocol that uses that encoding. Honestly not a very good name - it'd be like iTerm or Kitty naming their image protocol Base64. But it was originally named DECwriter Graphics (after the printer that first supported it), which is even worse IMO.

Anyway my point is that DRCS soft fonts are completely unrelated to the Sixel image protocol - they just happen to use the same binary-to-text encoding. So it's perfectly normal for a terminal to support one and not the other. Almost all DEC terminals supported some form of soft font, but few support the Sixel image protocol.

As for terminal emulators supporting DRCS, the ones I've personally tested successfully include KoalaTerm, MLTerm, PowerTerm, Reflection Desktop, RLogin, VTStar, and Windows Terminal. I believe Kermit95 and Contour have also recently added support, although I haven't tested either of them yet.

I'm fairly certain Mintty doesn't support DRCS, and I don't see any sign of extension 7 in their DA1 report. I'm not sure about AbsoluteTelnet or SecureCRT.

But feel free to ignore this if you're not interested. I was just throwing it out as an idea in case anyone felt inspired to give it a try.

I always appreciate your thoughts @j4james, I recorded all DA1 response from mintty and others by direct interrogation/use, and, it is not congruent with the source code of mintty. I've been looking at it for quite some time, seems to be a bug with windows conhost.exe overrides, but it appears to my Windows 11 machine that conhost.exe/ConPTY overrides the DA1 response,

mintty matches the DA1 response of your commit from 2023 https://github.com/microsoft/terminal/blob/3c3b1aac02320f49f6887f08e6d6d1d56132b1ef/src/terminal/adapter/adaptDispatch.cpp#L1500 on 10.0.26100.8328 (Windows 11 24H2).

From mintty:

Python 3.14.3 (tags/v3.14.3:323c59a, Feb  3 2026, 16:04:56) [MSC v.1944 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import blessed
>>> t=blessed.Terminal()
>>> list(sorted(t.get_device_attributes().extensions))
[6, 7, 21, 22, 23, 24, 28, 32, 42]
>>>

OK, I see your problem now. If you're testing a Windows-based terminal that's dependent on conpty, it needs to be using a modern pass-through version of the library, otherwise you're just testing conpty itself. It's like trying to test the terminal's functionality from within tmux.

With mintty, it used to be possible to bypass conpty (at least for running WSL) by using a wsltty build, but that apparently changed in version 3.8.0, so I think it's also limited by conpty now. And they don't really support using a modern version of conpty - their suggested solution is to replace the system libraries, which I wouldn't recommend.

Other Windows-based terminals are generally better, but if they have a Linux version, it's easier to just test on Linux. And terminals with a built-in ssh or telnet are also fine, because then you're connecting directly to your test system, and conpty isn't a factor.

I do use an arch linux docker container and Xvfb to test a majority of terminals, but of course mintty isn't available there.

That I am testing conpty instead of a native emulator's true capabilities is precisely the point of ucs-detect, to test the software as it is distributed and not theoretical capabilities if it's build and installation were customized

I'll just make a note about the unreliable results of mintty and other CONPTY-based terminals in the ucs-detect documentation. jquast/ucs-detect#52

I will test/integrate with --cpr-sync, I've been ignoring it so far, in fact looks like it freezes up the video for some reason :)

Do you know how I could reproduce this? I'd like to investigate this in order know if I can enable CPR sync by default.

When very busy with encoding graphics at scale as high as 11x, CPU usage of gambaterm's video encoding can also be as high as 70%, bandwidth as high as 3,000KB/s.

Oh I didn't realize that. Let me try to summarize:

• Octants:
  • low bandwidth
  • low CPU usage
  • requires small terminal windows
  • leaky colors
  • not widely and cannot be detected
• Stretch sextants
  • low bandwidth
  • low CPU usage
  • requires a medium terminal windows
  • leaky colors
  • widely supported
• Half-blocks
  • medium bandwidth
  • low CPU usage
  • requires a large terminal window
  • no leaky colors
  • widely supported
• Sixels
  • medium bandwidth that increases with scaling
  • heavy on the CPU at large scale
  • can adapt to all terminal window size
  • rarely supported
• Blitless sixels
  • high bandwidth that increases with scaling
  • heavy on the CPU at large scale
  • can adapt to all terminal window size
  • sometimes supported

And then I'm not sure what to write for kitty, did you manage to use compression in a way so that the required bandwidth do not increase with scaling?

The only issue is that it would be really hard to communicate to the user, "if you make your window large enough, and font small enough, it will improve the FPS and lower bandwidth, or, that If you make your font smaller, it will switch to text mode and the picture will get larger ?!

You meant "if you make your window small enough it will improve the FPS and lower the bandwidth"? Because a small window means small scaling, right?

Something like this:

But yes that's my main concern, especially because we first need to detect which rendering method the terminal support in order to provide users with their available options; I'm not sure that a README would really help here.

Well I realize now that the “click” and “pops” I’m hearing especially with high pitched square wave audio is not an artifact of “cpu usage too high causing dropped audio frames”, rather an effect of the resampling/audio pipeline. I ended up spending a lot of time looking at the wrong place.

I’m working on rolling back all of the extra options and I’ll round it out for review in a few days, frame dropping works fine, graphics work fine, even at largest scale with —cpr-sync.

I have been working on carving larger graphics images for kitty protocol into “bands” of ~256K each, so that the rendering pipeline never encodes a full frame but just pieces at a time, and yielding until next frame anytime the audio buffer is detected under 20% full (it doesn’t know, it just projects according to time elapsed, a sort of “time budget” for the renderer)

although it causes a small amount of “tearing” on very busy frames it’s much smoother during that process, usually there are some audio clicks and visual jitter here that this banding fixes, I’m doing profiling and debugging on that to see if the complexity is worth it and I’ll share when complete.

sizel has a kind of scaling built in that is linear, so 1x vs. 11x scaling is about the same bandwidth there, and so sixel is preferred over kitty when both are supported. Kitty on the other hand is bilinear scaling, I’ve given up trying to use the protocol’s scaling, even with “pixel padding” to ensure scaling is done at exact integers it is still blurry.

so anyway it seems all graphic scales work really well locally, and, we can set an optional bandwidth limit for servers, which drops frames when busy to limit it, still working out the “banding” issues but it appears it also reduces total KB/s because it no longer tries to encode and send whole-screen images but only partial, slightly lower fidelity but these are busy screen like when “sliding” to new screens in Zelda